Every figure on this page is transcribed from the training notebook shipped with the project — the architecture, dataset counts and the exact scoring rules. Nothing is estimated.
Audio
7.5MCNN + BiLSTM + RNN over log-Mel spectrograms
Video
33.3MResNet3D r3d_18 over 16-frame clips
Fusion
0.2MMLP over the 568-dim fused vector
What a submitted video walks through before a result appears.
ffmpeg → 16 kHz mono WAV
128-band log-Mel → CNN + BiLSTM + RNN
16 frames @ 112 px → ResNet3D r3d_18
568-dim vector → Fusion MLP
Softmax → rule-based indicator scores
What each stage is built from and the signal it consumes.
CNN + BiLSTM + RNN over log-Mel spectrograms
Layers
Specifications
ResNet3D r3d_18 over 16-frame clips
Layers
Specifications
MLP over the 568-dim fused vector
Layers
Specifications
Trained and evaluated on paired audio/video recordings of acted emotional speech and song.
RAVDESS
Speech + Song, paired audio/video
The fusion softmax is turned into four indicators by a transparent rule layer — no second model, no learned weights, so every score can be traced back to the emotion probabilities above.
High sadness together with low happiness. The multiplication means both conditions must hold before the score rises.
score = sad × (1 − happy)
Driven by fear, with a small reinforcement from surprise.
score = 0.85 × fearful + 0.15 × surprised
Driven by anger, reinforced by fear.
score = 0.6 × angry + 0.4 × fearful
A dominant, flat neutral affect across the recording.
score = neutral
No indicator is named unless it scores at least 15%. Below that the emotional pattern is reported as stable.
This result is not a medical diagnosis. Please consult a professional.
The environment the published metrics were produced on, and the settings each stage was trained with.
The same three networks, scored live on a two-minute video.
Get Started
English
العربية